Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Perform retries for certain HTTP status codes when submitting metrics to Datadog API #1359

Merged
merged 1 commit into from
Oct 11, 2024

Conversation

thyandrecardoso
Copy link
Contributor

This proposes adding some retries when Datadog responds with certain status codes. We're using Datadog API reporter and it is frequent to have some transient failures, which is the motivation for these changes.

I've also considered dealing with some Failures but, besides making testing harder, it is also not clear if those correspond to situations where the server did not receive the request at all and, thus, we might be duplicating data.

So, having retries only for the success case of the request and work with HTTP status codes seemed more sensible.

What do you think, does this make sense?

@ivantopo
Copy link
Contributor

ivantopo commented Sep 2, 2024

Yeap, I think the change makes sense. My only concern is whether the list of status codes that can be retried should be expanded/reduced. Is there any official Datadog documentation about when a client could/should retry requests?

@thyandrecardoso
Copy link
Contributor Author

thyandrecardoso commented Sep 2, 2024

Well, I found these:

I was unable to find anything in the official docs though :(

Nonetheless, the current list in this PR looks okish. We can also move these retriable status codes into configuration, but as we only have 1 endpoint and 1 verb being used against the public api, seems unlikely we will ever want to tweak those... 🤔

Copy link
Contributor

@hughsimpson hughsimpson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm :shipit:

@hughsimpson hughsimpson merged commit d9f7f9f into kamon-io:master Oct 11, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants